Tagging people who might be interested in weighing in:

@lehins @WhatisRT @zliu41 @effectfully @MicroProofs @colll78 @ana-pantilie @Unisay @Quantumplation @rvcas

@effectfully With respect to your reply here about intersectionValue and flattenValue:

I agree with you on flattenValue just being valueData. I usually only use aiken's flatten for pattern matching and valueData covers that use case.

However, for using intersection for lookupTokens, I use aiken's tokens a lot in my CIP-89 protocols so the extra overhead would quickly add up. I'm not a software engineer, but IMO lookupTokens is just a simple key lookup so I don't see there being much of a burden from supporting it as a separate builtin. How is it much different than lookupCoin?

We'll need mkEmptyValue as a primitive ...

I had assumed this was covered by unValueData $ List []. What would valueData return if used on an empty BuiltinValue?

What's the motivation for negateValue?

Subtracting values. unionValue only adds two values. I can use negateValue on one and then unionValue will do subtraction. This is useful for calculating the change to return at runtime.

... generalize policies to return [(CurrencySymbol, [(TokenName, Integer)])]?

policies is meant to only return the policy ids in the BuiltinValue like aiken's version. It is effectively just the keys function from Data.Map in Haskell. What you are suggesting just seems like valueData which would be more costly since it is processing extra unneeded information.

What's the motivation for intersection?

I may have gotten ahead of myself on this one. The idea was to have a single builtin support arbitrary higher-level filtering functions like aiken's restricted_to. But I'm not sure now if it will actually work the way I was thinking...

I had assumed this was covered by unValueData $ List []. What would valueData return if used on an empty BuiltinValue?

It's actually unValueData $ Map []. But that's only the case for Plutus V1 through V3. From V4, there will be a new Value constructor for Data, so Value will no longer be encoded as a Map. That said, I don't think emptyValue is actually necessary since we can just use a constant, so I take it back.

What you are suggesting just seems like valueData

This is also only the case for Plutus V1 through V3. From V4, valueData will essentially be a no-op: just wrapping a Value within Data's Value constructor.

The broader problem with Value is this: there are a lot of useful operations on nested maps, any subset of them can be made primitives, and the possibilities are almost endless. So I'd want to proceed very carefully, and give people enough time to think it through and reach consensus that it is the right primitive.

For example, if we have a valueToList primitive that returns [(CurrencySymbol, [(TokenName, Integer)])], then many other primitives can be done without primitives. So to make a case that a particular primitive should be a primitive, it will be very helpful to, for example, lay out some concrete use cases where the primitive leads to significant performance improvements.

I believe I made a huge mistake not including negateValue in the original CIP. negateValue is required for a huge number of use-cases of unionValue (to subtract one value from another). Can anyone evaluate the scope of this builtin? Is it possible we could introduce it? It is undoubtably the single most critical value builtin I neglected.

Without negateValue, the implementation of the programmable tokens CIP will not be possible, so this is very high priority for me.

Even if we need time to evaluate the demand and consider alternatives for some of the value operations proposed in this CIP, I believe negateValue is a special case where the need for it is very clear, and we have very high confidence that this is indeed a primitive value operation that we will need and it is very general.

@colll78 in Plutus V1 through V3 the builtin Value will be encoded as a Map in Data, so negateValue (and indeed anything else) can be done without a builtin. So even if it doesn't make it into the intra-era HF, this shouldn't be a deal breaker for most applications, should it? We can certainly add something like negateValue for V4.

As you can see, there's a suggestion above regarding generalizing negateValue into scaleValue. So it may take some time to decide what the best option is, even though the implementation of negateValue itself may not be difficult. It's also not clear whether there are more options.

@colll78 in Plutus V1 through V3 the builtin Value will be encoded as a Map in Data, so negateValue (and indeed anything else) can be done without a builtin. So even if it doesn't make it into the intra-era HF, this shouldn't be a deal breaker for most applications, should it? We can certainly add something like negateValue for V4.

As you can see, there's a suggestion above regarding generalizing negateValue into scaleValue. So it may take some time to decide what the best option is, even though the implementation of negateValue itself may not be difficult. It's also not clear whether there are more options.

negateValue can indeed be done without a builtin, but it is prohibitively expensive. In collaboration with IOG I have been working to introduce programmable assets to Cardano (https://github.com/input-output-hk/wsc-poc), the framework itself is complete, all features are implemented, and it is fully functional; the only reason that the feature is not live in production for end users is that the contracts are prohibitively expensive for mainnet production use. I have done an extensive amount of benchmarking, and the bottleneck is the value operations, particularly unionValue and negateValue, these are the two most expensive operations currently. The issue with the non-builtin implementation of negateValue is that it must traverse the entire value structure:

-- | Negate the quantity of every token in a Value
negateValue :: BuiltinData -> BuiltinData 
negateValue = BI.mkMap . negateOuter . BI.unsafeDataAsMap
 where
  negateOuter  value =
    BI.caseList'
      BI.mkNilData                        
      (\pair restCS ->                                 
        let newPair =
              BI.casePair pair $ \(cs, tkMapData) ->
                BI.mkPairData cs (negateTokens $ BI.unsafeDataAsMap tkListData)
         in BI.mkCons newPair (negateOuter restCS)
      )
    value
  negateTokens tks =
      BI.caseList'
        BI.mkNilData                        
        (\tkPair restTk ->
            let newTkPair =
                  BI.casePair tkPair $ \(tkName, amtData) ->
                     BI.mkPairData tkName (BI.mkI $ 0 - (BI.unsafeDataAsI amtData))
             in BI.mkCons newTkPair (negateTokens restTk)
        )
        tks

The above blows up the budget when applied to even eight or nine values with less than ten unique tokens each. This alone makes it impossible for programmable tokens to go to production.

caseList' and casePair currently expand to large and inefficient terms: https://github.com/IntersectMBO/plutus/blob/7ebd44f7c83ba4909a66ab3466802ad6b9714fd2/plutus-tx-plugin/src/PlutusTx/Compiler/Builtins.hs#L390-L527

After the HF, the terms under the _BuiltinCasing case will be used, which should be a lot cheaper. Of course, they may still not be cheap enough for your use case.

Based on the discussion it seems sensible to implement scaleValue for the intra era HF, so we'll do that. @colll78 @fallen-icarus if you can add scaleValue to the original CIP, that would be helpful, since it makes it clear what the exact scope for the intra era HF will be. Making a separate CIP just for scaleValue is fine, too.

I'm fine with scaleValue being added to CIP-153. Using a new CIP for scaleValue seems like a bad documentation practice. I'll update this CIP once CIP-153 has been updated.

This CIP isn't an update PR. It is a brand new CIP. AFAIU CIP-153 is for the builtins that will be added in the next intra-era hardfork. Meanwhile, this CIP is for the builtins that will be added after that hardfork, but for the new era.

Thanks @fallen-icarus... I had a feeling I was missing something. I was a little disoriented because all CIP-0153 updates have currently been merged so I didn't know what updates you might be talking about. Therefore, as I understand it, you're already free to submit your own CIP-0153 update adding scaleValue (please correct again if that seems wrong).